Skip to content

Fix Arrow chunk download telemetry error bucketing - #1649

Open
prathamesh-db wants to merge 3 commits into
databricks:mainfrom
prathamesh-db:codex/fix-arrow-chunk-error-bucketing
Open

Fix Arrow chunk download telemetry error bucketing#1649
prathamesh-db wants to merge 3 commits into
databricks:mainfrom
prathamesh-db:codex/fix-arrow-chunk-error-bucketing

Conversation

@prathamesh-db

@prathamesh-db prathamesh-db commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Description

An internal telemetry investigation found Arrow chunk download failures being emitted as DOWNLOAD_FAILED, which is an internal lifecycle state rather than a stable driver error code.

This change emits the existing CHUNK_DOWNLOAD_ERROR code for genuine download failures, preserves processing failures after download succeeds, and rethrows an existing typed DatabricksSQLException when a waiting chunk provider receives it. Untyped readiness failures still use CHUNK_READY_ERROR, and timeouts retain the original TimeoutException as their cause.

The change improves telemetry bucketing and exception preservation; it does not change download or retry behavior.

Testing

Focused tests cover successful download, body-read failure bucketing, malformed Arrow processing, typed future failures, untyped readiness failures, and readiness timeouts. The repository-wide Spotless check also passes.

Telemetry Errors

  • Not applicable — this PR does not add or change a telemetry-visible error.
  • Applicable — the error uses DatabricksDriverErrorCode where appropriate, and any
    new code is uniquely numbered and tested.
  • Applicable — maintainer help is requested to confirm that the existing
    CHUNK_DOWNLOAD_ERROR and preserved processing-error classifications remain appropriate.

Additional Notes to the Reviewer

This intentionally reuses existing error codes; no new error enum is needed.

@prathamesh-db
prathamesh-db marked this pull request as ready for review August 21, 2026 05:20
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
@prathamesh-db
prathamesh-db force-pushed the codex/fix-arrow-chunk-error-bucketing branch from 5c8201d to 93bff53 Compare August 25, 2026 10:09
Signed-off-by: Prathamesh Baviskar <prathamesh.baviskar@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant